Ordered memory pages transmission in virtual machine live migration

ABSTRACT

Systems and methods for virtual machine live migration. An example method may comprise: identifying, by a first computer system executing a virtual machine undergoing live migration to a second computer system, a plurality of stable memory pages comprised by an execution state of the virtual machine, wherein the plurality of stable memory pages comprises memory pages that have not been modified within a defined period of time; transmitting the plurality of stable memory pages to the second computer system; determining that an amount of memory comprised by a plurality of unstable memory pages is below a threshold value, wherein the plurality of unstable memory pages comprises memory pages that have been modified within the defined period of time; and transmitting the plurality of unstable memory pages to the second computer system.

TECHNICAL FIELD

The present disclosure is generally related to virtualized computersystems, and is more specifically related to systems and methods forvirtual machine live migration.

BACKGROUND

Virtualization may be viewed as abstraction of some physical componentsinto logical objects in order to allow running various software modules,for example, multiple operating systems, concurrently and in isolationfrom other software modules, on one or more interconnected physicalcomputer systems. Virtualization allows, for example, consolidatingmultiple physical servers into one physical server running multiplevirtual machines in order to improve the hardware utilization rate.Virtualization may be achieved by running a software layer, oftenreferred to as “hypervisor,” above the hardware and below the virtualmachines. A hypervisor may run directly on the server hardware withoutan operating system beneath it or as an application running under atraditional operating system. A hypervisor may abstract the physicallayer and present this abstraction to virtual machines to use, byproviding interfaces between the underlying hardware and virtual devicesof virtual machines. Processor virtualization may be implemented by thehypervisor scheduling time slots on one or more physical processors fora virtual machine, rather than a virtual machine actually having adedicated physical processor. Memory virtualization may be implementedby employing a page table (PT) which is a memory structure translatingvirtual memory addresses to physical memory addresses.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure is illustrated by way of examples, and not by wayof limitation, and may be more fully understood with references to thefollowing detailed description when considered in connection with thefigures, in which:

FIG. 1 depicts a high-level component diagram of an illustrative examplecomputer system 1000 in accordance with one or more aspects of thepresent disclosure;

FIGS. 2-3 depict flow diagrams of example methods for virtual machinelive migration, in accordance with one or more aspects of the presentdisclosure; and

FIG. 4 depicts a block diagram of an illustrative computer systemoperating in accordance with the examples of the present disclosure.

DETAILED DESCRIPTION

Described herein are methods and systems for ordered transmission ofmemory pages in virtual machine live migration. “Live migration” hereinrefers to the process of moving a running virtual machine from an originhost computer system to a destination host computer system withoutdisrupting the guest operating system and/or the applications executedby the virtual machine. In certain implementations, a migration agentmay copy the execution state of the virtual machine being migrated,including a plurality of memory pages, from the origin host to thedestination host while the virtual machine is still running at theorigin host. Upon completing the memory copying operation, the migrationagent may re-copy the memory pages which have been modified, or became“dirty,” during the memory copying operation. The migration agent mayiteratively repeat the copying of dirty memory pages, until apre-defined convergence condition has been met, e.g., until the memorypages “dirtying” rate becomes less than or equal to the virtual machinestate transfer rate. When the convergence condition is met, the virtualmachine may be stopped at the origin host, the dirty memory pages may becopied to the destination host, and the virtual machine may be restartedat the destination host.

The ratio of the total size of dirty memory pages which need to beretransmitted to the overall amount of the virtual machine memory can beindicative of the overall efficiency of the migration process. To reducethe number of the retransmitted memory pages, a migration agent mayobserve the virtual machine behavior and transmit the memory pages in aparticular order, as described in more details herein below. In anillustrative example, the migration agent may cause stable memory pages(e.g., memory pages that have not been modified within a defined periodof time) to be transmitted first, followed by the transmission ofremaining memory pages.

Various aspects of the above referenced methods and systems aredescribed in details herein below by way of examples, rather than by wayof limitation.

FIG. 1 depicts a high-level component diagram of one illustrativeexample of a computer system 1000 in accordance with one or more aspectsof the present disclosure. The computer system 1000 may include computersystems 110 and 120 interconnected via a network 130. Each of the hostcomputer systems 110 and 120 may include one or more processors 131communicatively coupled to memory devices 133 and input/output (I/O)devices 135. Local connections within each of the hosts 110 and 120,including the connections between processors 131, memory 133, and I/Odevices 135 may be provided by one or more local buses 150 of suitablearchitecture.

“Processor” herein shall refer to a device capable of executinginstructions encoding arithmetic, logical, or I/O operations. In oneillustrative example, a processor may follow Von Neumann architecturalmodel and may include an arithmetic logic unit (ALU), a control unit,and a plurality of registers. In a further aspect, a processor may be asingle core processor which is typically capable of executing oneinstruction at a time (or process a single pipeline of instructions), ora multi-core processor which may simultaneously execute multipleinstructions. In another aspect, a processor may be implemented as asingle integrated circuit, two or more integrated circuits, or may be acomponent of a multi-chip module (e.g., in which individualmicroprocessor dies are included in a single integrated circuit packageand hence share a single socket). A processor may also be referred to asa central processing unit (CPU). “Memory device” herein shall refer to avolatile or non-volatile memory device, such as RAM, ROM, EEPROM, or anyother device capable of storing data. “I/O device” herein shall refer toa device capable of providing an interface between one or more processorpins and an external device capable of inputting and/or outputtingbinary data.

Host computer systems 110 and 120 may run a plurality of virtualmachines 112 and 122, by executing hypervisors 115 and 125,respectively. Each of hypervisors 115 and 125 may abstract the physicallayer, including processors, memory, and I/O devices, and present thisabstraction to virtual machines 112, 122 as virtual devices, includingvirtual processors, virtual memory, and virtual I/O devices.

Virtual machine 112, 122 may execute a guest operating system which mayutilize the underlying virtual devices, including virtual processors,virtual memory, and virtual I/O devices. One or more applications may berunning on virtual machine 112 under the guest operating system.

As schematically illustrated by FIG. 1, host computer systems 110 and120 may be communicatively coupled to a host controller 105 which mayreside on a designated computer system or on one of the hosts 110, 120.Host controller 105 may manage a plurality of virtual machines,including virtual machines 112 and 122. Host controller 105 may create avirtual machine, delete a virtual machine, and perform other virtualmachine management functions. In certain implementations, hostcontroller 105 may include a migration agent 107 which may manage livemigration of virtual machines between host computer systems 110 and 120in accordance with one or more aspects of the present disclosure. Incertain implementations, host computer systems 110 and 120 may includehost migration agents 117 and 127, respectively, designed to perform atleast some of the virtual machine migration management functions inaccordance with one or more aspects of the present disclosure.

For illustrative purposes, host computer system 110 may be referred toas the origin host from which virtual machine 140 may migrate to hostcomputer system 120, which may accordingly be referred to as thedestination host. Live migration may involve copying the virtual machineexecution state from the origin host to the destination host. Thevirtual machine execution state may comprise the memory state, thevirtual processor state, the virtual devices state, and/or theconnectivity state.

In certain implementations, host migration agent 107, 117, and/or 127may copy, over network 130, the execution state of the migrating virtualmachine 140, including a plurality of memory pages, from origin host 110to destination host 120 while virtual machine 140 is still running onorigin host 110. Upon completing the virtual machine state copyingoperation, the migration agent may re-copy the memory pages which havebeen modified, or became “dirty,” during the state copying operation.The migration agent may iteratively repeat the copying of dirty memorypages, until a pre-defined condition has been met, e.g., until thememory pages “dirtying” rate becomes less than or equal to the virtualmachine state transfer rate. When the convergence condition is met,virtual machine 140 may be stopped at origin host 110, the dirty memorypages may be copied to destination host 120, and virtual machine 140 maybe restarted at destination host 120.

As noted herein above, the migration agent may reduce the number ofmemory pages that need to be iteratively retransmitted, by observing thevirtual machine behavior and transmitting the memory pages in aparticular order. In an illustrative example, the migration agent mayidentify a plurality of stable memory pages (e.g., memory pages thathave not been modified within a defined period of time) of the executionstate of the virtual machine being migrated, and transmit the identifiedmemory pages to the destination host computer system. The operations ofidentifying stable memory pages and transmitting them to the destinationhost may be repeated until the total size of the remaining memory pagesfalls below a specified threshold, at which stage the remaining(“unstable”) memory pages may be transmitted to the destination hostcomputer system in an arbitrary order. Upon transmitting the unstablememory pages, the virtual machine may be stopped at the origin host, andthe dirty pages (i.e., the pages that have been modified since beingtransmitted to the destination host) may be identified andre-transmitted to the destination host.

To identify the stable memory pages, the origin host may enable memoryaccess tracking and recurrently, at a defined memory tracking interval,identify a plurality of memory pages that have been modified within thelast memory scanning cycle. When the total size of newly modified memorypages (i.e., memory pages that have been added to the plurality ofmodified memory pages during the current memory scanning cycle) fallsbelow a pre-defined value, the remaining memory pages may be declared“stable.” Stable memory pages are those that have not been modifiedwithin a defined time period, and hence may be assumed not to bemodified for a comparable time period immediately following the timeperiod of being stable. Thus, in order to decrease the ratio of memorypages that would need to be re-transmitted as having been modified afterthe previous transmission, the stable memory pages may be transmitted tothe destination memory system before the memory pages that have recentlybeen modified.

Example methods of virtual machine live migration in accordance with oneor more aspects of the present disclosure are described herein belowwith references to flow diagrams of FIGS. 2 and 3.

FIG. 2 depicts a flow diagram of an illustrative example of a method 200for ordered transmission of memory pages during virtual machine livemigration. Method 200 and/or each of its individual functions, routines,subroutines, or operations may be performed by one or more processors ofthe computer system (e.g., host computer system 110 of FIG. 1) executingthe method. In certain implementations, method 200 may be performed by asingle processing thread. Alternatively, method 200 may be performed bytwo or more processing threads, each thread executing one or moreindividual functions, routines, subroutines, or operations of themethod. In an illustrative example, the processing threads implementingmethod 200 may be synchronized (e.g., using semaphores, criticalsections, and/or other thread synchronization mechanisms).Alternatively, the processing threads implementing method 200 may beexecuted asynchronously with respect to each other.

At block 210, the host computer system may initialize various variablesused by the method (e.g., the timeout to be used in determining whetherthe method should be terminated, as described herein below withreferences to block 260).

At block 220, the origin host computer system may identify stable memorypages, as described in more details herein below with references to FIG.3. The identified stable memory pages may be transmitted to thedestination host.

Responsive to ascertaining, at block 230, that the total size or numberof the remaining memory pages does not exceed a pre-defined ordynamically calculated threshold value, the processing may continue atblock 240. In an illustrative example, the threshold value may be chosenas a function of the total size of the virtual machine's memory and/orthe virtual machine state transfer rate. The latter may be defined asthe amount of data that may be transmitted, or have been actuallytransmitted, from the origin host to the destination host over thenetwork in a unit of time. Responsive to determining, at block 230, thatthe total size or number of the remaining memory pages exceeds thethreshold value, the method may branch to block 260.

Alternatively, at block 230 the method may determine that the rate ofchange of the remaining memory pages is below a threshold value, whichmay be chosen as a function of the total size of the virtual machine'smemory and/or the virtual machine state transfer rate. Thus, responsiveto determining, at block 230, that the rate of change of the remainingmemory pages is below the threshold value, the method may branch toblock 260.

Responsive to determining, at block 260, that a pre-defined timeout hasnot yet expired, the method may loop back to identifying a set of stablememory pages to be transmitted to the destination host.

At block 240, having determined that the total size or number ofunstable memory pages has fallen below the pre-defined threshold value,the host computer system may transmit the unstable memory pages to thedestination host.

The migration agent may iteratively repeat the memory page transmissionoperations referenced by blocks 210-240, until a pre-defined convergencecondition has been met, e.g., until the memory pages “dirtying” ratebecomes less than or equal to the virtual machine state transfer rate.Responsive to determining, at block 250, that the convergence conditionhas been met, the method may terminate. When the convergence conditionis met, the virtual machine may be stopped at the origin host, the dirtymemory pages (i.e., the pages that have been modified since beingtransmitted to the destination host) may be copied to the destinationhost, and the virtual machine may be restarted at the destination host.

Otherwise, responsive to determining, at block 250, that the convergencecondition has not been met, the method may loop back to identifying aset of stable memory pages to be transmitted to the destination host, asdescribed herein above with references to block 210. In certainimplementations, the migration agent may declare the migration processfailure should the virtual machine state fail to stabilize within apre-defined time period. Alternatively, the migration agent may switchto a post-copy migration method, by stopping the virtual machine,transferring a subset of the virtual machine execution state (includingthe virtual processor state and non-pageable memory state) to thedestination host, resuming the virtual machine at the destination host,generating a page fault responsive to detecting the virtual machine'sattempt to access a memory page which has not yet been transferred, andtransferring the page from the origin host to the destination hostresponsive to the page fault.

The timeout to be evaluated at block 260 is designed to prevent themethod from going into an indefinite loop. Should the timeout expirewhile the total size or number of unstable memory pages has failed tofall below the pre-defined convergence threshold, the host computersystem may remedy the situation by modifying one or more methodparameters or switching to another migration method. In an illustrativeexample, the host computer system may increase the threshold size ofunstable memory pages upon reaching which the method switches totransmitting the remaining unstable memory pages, as described withreferences to block 230. In another illustrative example, the hostcomputer system may increase the timeout and loop back to identifying aset of stable memory pages to be transmitted to the destination host, asdescribed herein above with references to block 220. Alternatively,responsive to determining, at block 260, that the timeout has expired,the host computer system may switch to a different migration procedure(e.g., to a procedure that involves transmitting the memory pages in arandom or arbitrary selected order).

FIG. 3 depicts a flow diagram of an illustrative example of a method 300for identifying stable memory pages belonging to memory address spacesof one or more virtual machines being migrated, in accordance with oneor more aspect of the present disclosure. Method 300 and/or each of itsindividual functions, routines, subroutines, or operations may beperformed by one or more processors of the computer system (e.g., hostcomputer system 110 of FIG. 1) executing the method. In certainimplementations, method 300 may be performed by a single processingthread. Alternatively, method 300 may be performed by two or moreprocessing threads, each thread executing one or more individualfunctions, routines, subroutines, or operations of the method. In anillustrative example, the processing threads implementing method 300 maybe synchronized (e.g., using semaphores, critical sections, and/or otherthread synchronization mechanisms). Alternatively, the processingthreads implementing method 300 may be executed asynchronously withrespect to each other.

At block 310, the origin host computer system may enable tracking ofmemory pages comprised by address spaces of one or more virtual machinesbeing migrated to a destination host computer system.

At blocks 320-330, the host computer system may wait until a trackingdelay is expired. The tracking delay value may be arbitrarily chosen ormay be calculated as a function of one or more variables, including,e.g., virtual machine total memory size. In certain implementations, thevalue may be set to zero, thus introducing no delay between successivememory scanning cycles described herein below with references to blocks350-360.

In an illustrative example, the processing thread which executes method300 may transition into a sleeping state for the duration of thetracking delay, thus yielding the processor to other processing threads.

Responsive to determining, at block 330, that the tracking delay hasexpired, the host computer system may, at block 340, scan the virtualmachine memory to identify a plurality of memory pages that has beenmodified since the last memory scanning iteration.

Responsive to determining, at block 350, that the total size or numberof the modified memory pages is less than a threshold value, the hostcomputer system may, at block 360, declare the remaining memory pages as“stable,” and the method may terminate. In an illustrative example, thethreshold value may be chosen as a function of the total size of thevirtual machine's memory and/or the virtual machine state transfer rate.

Otherwise, if the total size or number of the modified memory pagesexceeds the threshold value (i.e., the set of modified memory pages hasat the last memory scanning iteration grown by more than the thresholdvalue), the method may, at block 370, evaluate a pre-defined timeout.Responsive to ascertaining that the timeout has not yet expired, themethod may loop back to the next memory scanning iteration referenced byblocks 320-340.

Before performing the next memory scanning iteration, the host computersystem may optionally disable tracking of modified memory pages, thusoptimizing the amount of memory to be tracked at the next iteration ofthe method. The host computer system may also optionally modify thetracking delay to be used in the next memory scanning iteration. Incertain implementations, the tracking delay may be re-calculated basedon the diminished size of the memory to be tracked. Alternatively, thetracking delay may be multiplied by a pre-defined or dynamicallycalculated positive factor being less than one. Upon re-calculating thetracking delay, the method may loop back to block 320.

The timeout to be evaluated at block 370 is designed to prevent themethod from going into an indefinite loop. Responsive to ascertaining,at block 370 that the timeout has expired and the set of modified memorypages has failed to stabilized, the host computer system may modify oneor more method parameters (e.g., the memory scanning timeout or thenewly modified pages size threshold) or declare the method failure.

The set of stable memory pages returned by method 300 may be used inconjunction of method 200, as described herein above with references toFIG. 2. The combination of the above described methods 200 and 300provides a heuristic-based method for efficient virtual machine livemigration.

FIG. 4 depicts an example computer system 1000 within which a set ofinstructions, for causing the computer system to perform any one or moreof the methods described herein, may be executed. In certainimplementations, computer system 1000 may correspond to the hostcomputer system 110, 120 of FIG. 1.

In an illustrative example, computer system 1000 may be connected (e.g.,via a network, such as a Local Area Network (LAN), an intranet, anextranet, or the Internet) to other computer systems. Computer system1000 may operate in the capacity of a server or a client computer in aclient-server environment, or as a peer computer in a peer-to-peer ordistributed network environment. Computer system 1000 may be provided bya personal computer (PC), a tablet PC, a set-top box (STB), a PersonalDigital Assistant (PDA), a cellular telephone, a web appliance, aserver, a network router, switch or bridge, or any device capable ofexecuting a set of instructions (sequential or otherwise) that specifyactions to be taken by that device. Further, the term “computer” shallinclude any collection of computers that individually or jointly executea set (or multiple sets) of instructions to perform any one or more ofthe methods described herein.

In a further aspect, the computer system 1000 may include a physicalprocessor 1002, a volatile memory 1004 (e.g., random access memory(RAM)), a non-volatile memory 1006 (e.g., read-only memory (ROM) orelectrically-erasable programmable ROM (EEPROM)), and a secondary memory1016 (e.g., a data storage device), which may communicate with eachother via a bus 1008.

Processor 1002 may be provided by one or more physical processors suchas a general purpose processor (such as, for example, a complexinstruction set computing (CISC) microprocessor, a reduced instructionset computing (RISC) microprocessor, a very long instruction word (VLIW)microprocessor, a microprocessor implementing other types of instructionsets, or a microprocessor implementing a combination of types ofinstruction sets) or a specialized processor (such as, for example, anapplication specific integrated circuit (ASIC), a field programmablegate array (FPGA), a digital signal processor (DSP), or a networkprocessor).

Computer system 1000 may further include a network interface device1022. Computer system 1000 also may include a video display unit 1010(e.g., an LCD), an alphanumeric input device 1012 (e.g., a keyboard), apointing device 1014 (e.g., a mouse), and an audio output device 1020(e.g., a speaker).

Secondary memory 1016 may include a non-transitory computer-readablestorage medium 1024 on which may be stored instructions of migrationagent 107, 117 implementing the methods for virtual machine livemigration described herein. Instructions of migration agent 107, 117 mayalso reside, completely or partially, within main memory 1004 and/orwithin processor 1002 during execution thereof by computer system 1000,hence, main memory 1004 and processor 1002 may also constitutemachine-readable storage media.

While computer-readable storage medium 1024 is shown in the illustrativeexample as a single medium, the term “computer-readable storage medium”shall include a single medium or multiple media (e.g., a centralized ordistributed database, and/or associated caches and servers) that storethe one or more sets of executable instructions. The term“computer-readable storage medium” shall also include any non-transitorymedium that is capable of storing or encoding a set of instructions forexecution by a computer that cause the computer to perform any one ormore of the methods described herein. The term “computer-readablestorage medium” shall include, but not be limited to, solid-statememories, optical media, and magnetic media.

The methods, components, and features described herein may beimplemented by discrete hardware components or may be integrated in thefunctionality of other hardware components such as ASICS, FPGAs, DSPs orsimilar devices. In addition, the methods, components, and features maybe implemented by firmware modules or functional circuitry withinhardware devices. Further, the methods, components, and features may beimplemented in any combination of hardware devices and softwarecomponents, or only in software.

Unless specifically stated otherwise, terms such as “updating”,“identifying”, “determining”, “sending”, “assigning”, or the like, referto actions and processes performed or implemented by computer systemsthat manipulates and transforms data represented as physical(electronic) quantities within the computer system's registers andmemories into other data similarly represented as physical quantitieswithin the computer system memories or registers or other suchinformation storage, transmission or display devices. Also, the terms“first,” “second,” “third,” “fourth,” etc. as used herein are meant aslabels to distinguish among different elements and may not necessarilyhave an ordinal meaning according to their numerical designation.

Examples described herein also relate to an apparatus for performing themethods described herein. This apparatus may be specially constructedfor the required purposes, or it may comprise a general purpose computersystem selectively programmed by a computer program stored in thecomputer system. Such a computer program may be stored in acomputer-readable non-transitory storage medium.

The methods and illustrative examples described herein are notinherently related to any particular computer or other apparatus.Various general purpose systems may be used in accordance with theteachings described herein, or it may prove convenient to construct morespecialized apparatus to perform the required method operations. Therequired structure for a variety of these systems will appear as setforth in the description above.

The above description is intended to be illustrative, and notrestrictive. Although the present disclosure has been described withreferences to specific illustrative examples, it will be recognized thatthe present disclosure is not limited to the examples described. Thescope of the disclosure should be determined with reference to thefollowing claims, along with the full scope of equivalents to which theclaims are entitled.

What is claimed is:
 1. A method, comprising: identifying, by a firstcomputer system executing a virtual machine undergoing live migration toa second computer system, a plurality of stable memory pages comprisedby an execution state of the virtual machine, wherein the plurality ofstable memory pages comprises memory pages that have not been modifiedwithin a defined period of time; transmitting the plurality of stablememory pages to the second computer system; determining that an amountof memory comprised by a plurality of unstable memory pages is below athreshold value, wherein the plurality of unstable memory pagescomprises memory pages that have been modified within the defined periodof time; and transmitting the plurality of unstable memory pages to thesecond computer system.
 2. The method of claim 1, further comprising:responsive to determining that an amount of memory comprised by theplurality of unstable memory pages exceeds the threshold value,repeating the identifying and the transmitting operations.
 3. The methodof claim 1, further comprising: determining that an amount of memorycomprised by the plurality of unstable memory pages exceeds thethreshold value; responsive to determining that a defined timeout hasexpired since initiating a virtual machine migration procedure,performing one of: modifying a parameter of the virtual machinemigration procedure or terminating the virtual machine migrationprocedure.
 4. The method of claim 1, wherein identifying the pluralityof stable memory pages comprises: enabling memory tracking by the firstcomputer system; identifying modified memory pages; and responsive todetermining that a number of the modified memory pages is less than adefined threshold value, identifying remaining memory pages of thevirtual machine as stable memory pages.
 5. The method of claim 4,wherein identifying modified memory pages is performed responsive toexpiration of a tracking delay.
 6. The method of claim 4, furthercomprising: responsive to determining that a number of the modifiedmemory pages exceeds the defined threshold value, repeating theidentifying modified memory pages operation.
 7. The method of claim 4,further comprising: responsive to determining that a number of themodified memory pages exceeds the defined threshold value, disablingtracking of the modified memory pages; and repeating the identifyingmodified memory pages operation.
 8. The method of claim 4, furthercomprising: responsive to determining that a number of the modifiedmemory pages exceeds the defined threshold value, updating the trackingdelay; and repeating the identifying modified memory pages operation. 9.The method of claim 1, wherein identifying the plurality of stablememory pages comprises: enabling memory tracking by the first computersystem; responsive to expiration of a tracking delay, identifyingmodified memory pages; and responsive to determining that a rate ofchange of the modified memory pages is less than a defined thresholdvalue, identifying remaining memory pages of the virtual machine asstable memory pages.
 10. A system of a first computer system,comprising: a memory; and one or more physical processors, coupled tothe memory, to: identify a plurality of stable memory pages comprised byan execution state of a virtual machine undergoing live migration to asecond computer system, wherein the plurality of stable memory pagescomprises memory pages that have not been modified within a definedperiod of time; transmit the plurality of stable memory pages to thesecond computer system; determine that an amount of memory comprised bya plurality of unstable memory pages is below a threshold value, whereinthe plurality of unstable memory pages comprises memory pages that havebeen modified within the defined period of time; and transmit theplurality of unstable memory pages to the second computer system. 11.The system of claim 10, wherein the processors are further to:responsive to determining that an amount of memory comprised by theplurality of unstable memory pages exceeds the threshold value, repeatthe identifying and the transmitting operations.
 12. The system of claim10, wherein the processors are further to: determine that an amount ofmemory comprised by the plurality of unstable memory pages exceeds athreshold value; responsive to determining that a defined timeout hasexpired since initiating a virtual machine migration procedure, performone of: modifying a parameter of the virtual machine migration procedureor terminating the virtual machine migration procedure.
 13. The systemof claim 10, wherein identifying the plurality of stable memory pagescomprises: enabling memory tracking by the first computer system;responsive to expiration of a tracking delay, identifying modifiedmemory pages; and responsive to determining that a number of themodified memory pages is less than a defined threshold value,identifying remaining memory pages of the virtual machine as stablememory pages.
 14. The system of claim 13, wherein the processors arefurther to: responsive to determining that a number of the modifiedmemory pages exceeds the defined threshold value, repeat the identifyingmodified memory pages operation.
 15. The system of claim 13, wherein theprocessors are further to: responsive to determining that a number ofthe modified memory pages exceeds the defined threshold value, disabletracking of the modified memory pages; and repeat the identifyingmodified memory pages operation.
 16. A computer-readable non-transitorystorage medium comprising executable instructions that, when executed bya first computer system, cause the first computer system to performoperations, comprising: identifying a plurality of stable memory pagescomprised by an execution state of a virtual machine undergoing livemigration to a second computer system, wherein the plurality of stablememory pages comprises memory pages that have not been modified within adefined period of time; transmitting the plurality of stable memorypages to the second computer system; determining that an amount ofmemory comprised by a plurality of unstable memory pages is below athreshold value, wherein the plurality of unstable memory pagescomprises memory pages that have been modified within the defined periodof time; and transmitting the plurality of unstable memory pages to thesecond computer system.
 17. The computer-readable non-transitory storagemedium of claim 16, further comprising executable instructions causingthe first computer system to perform operations, comprising: responsiveto determining that an amount of memory comprised by a plurality ofunstable memory pages exceeds a threshold value, repeating theidentifying and the transmitting operations.
 18. The computer-readablenon-transitory storage medium of claim 16, further comprising executableinstructions causing the first computer system to perform operations,comprising: determining that an amount of memory comprised by theplurality of unstable memory pages exceeds a threshold value; responsiveto determining that a defined timeout has expired since initiating avirtual machine migration procedure, performing one of: modifying aparameter of the virtual machine migration procedure or terminating thevirtual machine migration procedure.
 19. The computer-readablenon-transitory storage medium of claim 16, wherein identifying theplurality of stable memory pages comprises: enabling memory tracking bythe first computer system; responsive to expiration of a tracking delay,identifying modified memory pages; and responsive to determining that anumber of the modified memory pages is less than a defined thresholdvalue, identifying remaining memory pages of the virtual machine asstable memory pages.
 20. The computer-readable non-transitory storagemedium of claim 16, further comprising executable instructions causingthe first computer system to perform operations, comprising: responsiveto determining that a number of the modified memory pages exceeds thedefined threshold value, repeat the identifying modified memory pagesoperation.